Local AI, catalogued

Every model you can ollama run.

A researched, continually-derived reference for locally-runnable open models — parameters, licenses, hardware requirements and honest trade-offs, generated from compact source facts rather than copy-pasted marketing.

Beyond local models

Cloud & API model reference

Quick-reference model ID strings for the major hosted providers, company by company, each with a copy-paste code sample. Compiled July 2026 — provider model strings shift often, so always cross-check against live docs before hardcoding one into production.

RAG pipeline building blocks

LangChain document loaders reference

A lookup sheet for every commonly used LangChain document loader — install command, import, code and key arguments — for feeding local or hosted models with your own documents. Each card below has the exact install command it needs.

Quick selection guide

Common next step: splitting loaded docs

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
chunks = splitter.split_documents(docs)

Chunking, done right

LangChain text splitters reference

Every text splitter in LangChain — internal working, constructor arguments, code samples and production chunk-size guidance. Splitting is the step right after loading: good chunks make retrieval precise; bad chunks make even a great model hallucinate.

Quick selection guide

Production chunk size guide

Capability matrix

Common next step: embed + store the chunks

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_google_genai import GoogleGenerativeAIEmbeddings

# 1. Load
docs = PyPDFLoader("resume.pdf").load()

# 2. Enrich metadata
for d in docs:
    d.metadata["doc_type"] = "resume"

# 3. Split
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    add_start_index=True,
)
chunks = splitter.split_documents(docs)

# 4. Embed + store
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=GoogleGenerativeAIEmbeddings(model="gemini-embedding-2"),
)

Text → math, done right

LangChain embeddings reference

The step right after splitting: turning each chunk into a dense vector that captures meaning, not just keywords. Foundations, every major provider, constructor arguments and production tips — no vector database content here, just the embedding layer itself.

Foundations

Em

What is an embedding?

A fixed-length vector of floating-point numbers representing the meaning of text in high-dimensional space. Texts with similar meaning land close together geometrically, regardless of exact wording — this is what lets "car" and "automobile" match even with zero shared keywords.

"king" → [0.021, -0.443, 0.117, ..., 0.089]   # e.g. 1536 numbers

Ds

Dense vs sparse vs hybrid

Type	Representation	Example	Strength
Sparse	Mostly zeros	TF-IDF, BM25	Exact keyword matching
Dense	Compact, every dim meaningful	OpenAI, Gemini, BGE	Semantic matching
Hybrid	Weighted fusion of both	BM25 + dense rerank	Best of both — common in enterprise

Co

Cosine similarity

The default distance metric for text embeddings — measures the angle between two vectors, ignoring magnitude. Range −1 (opposite) to 1 (identical direction).

cosine_similarity(A, B) = (A · B) / (‖A‖ × ‖B‖)

import numpy as np
def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Dp

Dot product & Euclidean distance

If vectors are normalized (unit length), dot product ranks identically to cosine similarity — why many vector DBs default to it internally, it skips the division. Euclidean distance measures straight-line distance instead of angle; sensitive to magnitude, so less common for text unless pre-normalized.

dot_product(A, B)  = Σ(Aᵢ × Bᵢ)
euclidean(A, B)     = √Σ(Aᵢ - Bᵢ)²

Dm

Dimensions

Dimension count = length of the output vector. Higher dimensions capture more nuance but cost more storage/compute per comparison. Many modern models support Matryoshka Representation Learning (MRL) — truncating to fewer dimensions via a dimensions argument while preserving most of the quality.

Model	Dimensions

Nr

Normalization

Normalizing vectors to unit length (‖v‖ = 1) makes dot product equivalent to cosine similarity — many vector DBs assume normalized vectors for their fastest index types. Most provider SDKs (OpenAI, Gemini) already return normalized vectors by default — verify before double-normalizing.

import numpy as np
def normalize(vec):
    v = np.array(vec)
    return (v / np.linalg.norm(v)).tolist()

Which embedding model?

Provider comparison matrix

Production notes

Query vs document embeddings

Providers like Gemini, Cohere, BGE and E5 differentiate embedding queries from embedding documents, via a task_type/input_type param or a text prefix. Using the same call for both works, but underperforms the provider's intended asymmetric retrieval setup.

query_vec = query_embedder.embed_query(user_question)
doc_vecs  = doc_embedder.embed_documents(chunk_texts)

Batch, don't loop

Always batch instead of looping embed_query per chunk — batching cuts API round-trips dramatically (10x+ throughput on large corpora). Cache embeddings for unchanged content to avoid re-embedding on every ingestion run.

texts = [doc.page_content for doc in chunks]
vectors = embeddings.embed_documents(texts)

Never mix models

Pin exact model versions in production — embedding a new corpus with a different model version than your existing index causes silent quality degradation. Vectors from two different models are never directly comparable, even at the same dimension count.

The retrieval half of RAG

LangChain vector databases reference

Where embedded vectors actually live: purpose-built stores that answer "which vectors are closest to this query?" fast, at scale. Theory, every major database, the retriever layer built on top, and production scaling guidance.

Foundations

Vd

Why a vector database?

A plain Python list of vectors works for a few thousand documents. Beyond that, linear scan becomes too slow, and you lose native persistence, filtering, concurrent access, CRUD, sharding, and replication. A vector database stores embeddings alongside source text and metadata, purpose-built to find the closest stored vectors to a query vector — at scale, across millions to billions of entries.

Ds

Distance metrics

Metric	Intuition	Used by (default)

Rule of thumb: if vectors are normalized to unit length, cosine similarity and dot product rank identically — many DBs default to dot product for the extra speed.

An

Exact search vs ANN

	Exact (brute-force)	ANN
Accuracy	100%	~95–99% (tunable)
Speed at scale	Slow	Fast
Use case	Small collections (<50k vectors)	Production-scale collections

Exact search checks every vector — accurate but O(n). ANN trades a little accuracy for massive speed, mandatory past roughly 100k–1M vectors.

Ix

Indexing algorithms

Algorithm	How it works	Trade-off

Industry recommendation

Enterprise comparison matrix

The retrieval layer

Retriever — the universal interface

Every vector store converts to a standard Retriever interface — this is what you actually plug into a chain. search_type can be "similarity" (standard top-k), "mmr" (diversity-aware, avoids redundant near-duplicates), or "similarity_score_threshold" (only above a minimum score).

retriever = db.as_retriever(
    search_type="similarity",       # or "mmr", "similarity_score_threshold"
    search_kwargs={"k": 4}
)
relevant_docs = retriever.invoke("What is FluenzyAI's tech stack?")

Metadata filtering

Scope semantic search to a subset — filter syntax varies slightly per DB (Chroma dict filters, Qdrant payload filters, Pinecone metadata filters); always check the specific DB's operator support ($eq, $in, $gte).

results = db.similarity_search(
    "interview preparation tips",
    k=5,
    filter={"doc_type": "blog", "language": "en"}
)

Hybrid search

Combines dense (semantic) + sparse (keyword/BM25) scoring — catches both meaning matches and exact term matches (product codes, names) that pure semantic search can miss.

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25_retriever = BM25Retriever.from_documents(final_documents)
bm25_retriever.k = 5
vector_retriever = db.as_retriever(search_kwargs={"k": 5})

hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.4, 0.6],
)

Reranking

A first-pass retriever fetches a broad candidate set (e.g. top 20), then a reranker model re-scores them for precision, returning a smaller top-k (e.g. 5) — improves precision even when the base retriever already looks relevant.

from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank

compressor = CohereRerank(model="rerank-english-v3.0", top_n=5)
reranking_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=db.as_retriever(search_kwargs={"k": 20}),
)

Parent Document Retriever

Indexes small chunks for precise search, but returns their larger parent chunk/document for generation — balances retrieval precision with generation context.

from langchain.retrievers import ParentDocumentRetriever
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.storage import InMemoryStore

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)

retriever = ParentDocumentRetriever(
    vectorstore=db, docstore=InMemoryStore(),
    child_splitter=child_splitter, parent_splitter=parent_splitter,
)

Multi-vector & contextual compression

Multi-vector retrieval stores multiple representations per document (e.g. a summary vector + full chunk vector) — search the summary, return the full chunk. Contextual compression strips irrelevant sentences out of each retrieved chunk before sending to the LLM, cutting noise and token cost.

from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor

Production RAG (full pipeline)

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_anthropic import ChatAnthropic

# 1. Load + Split
docs = PyPDFLoader("resume.pdf").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120).split_documents(docs)

# 2. Embed + Store
embeddings = OpenAIEmbeddings(model="text-embedding-3-large", dimensions=1024)
db = QdrantVectorStore.from_documents(chunks, embeddings, url="http://localhost:6333", collection_name="resumes")

# 3. Hybrid Retriever
bm25 = BM25Retriever.from_documents(chunks)
vector_retriever = db.as_retriever(search_kwargs={"k": 5})
retriever = EnsembleRetriever(retrievers=[bm25, vector_retriever], weights=[0.3, 0.7])

# 4. Generate
llm = ChatAnthropic(model="claude-sonnet-5")
context_docs = retriever.invoke("candidate's key skills")
context = "\n\n".join(d.page_content for d in context_docs)
answer = llm.invoke(f"Context:\n{context}\n\nQuestion: What are the candidate's key skills?")

Scaling & enterprise best practices

Enterprise best practices

Match your DB choice to existing infra first (pgvector on Postgres, MongoDB Atlas on Mongo) before adding a new system.
Always version your collection alongside the embedding model — never mix vectors from two different models in one collection.
Use metadata filtering to scope multi-tenant or multi-source data rather than creating excessive tiny collections.
Start with similarity_search; add MMR/hybrid/reranking only once you observe a real precision problem — extra retrieval stages cost latency and money.
Load-test retrieval latency and recall at production-scale vector counts — behavior at 1k vectors doesn't predict behavior at 1M.
Back up vector data separately from your primary DB if it isn't already durable — some ANN indexes are expensive to rebuild.